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FOREWORD 



This crninnphrif) af\6 guidance program senes ts paiterned 
after thf^ Pprtormancc Baft<«d Teacher Eclucation modules 
(lesi^nec] ancl (levelopeciat the National Center tor Research 
m Vocational Education under federal Number NE-COO 3-77 
Beca*use this model has been successtully and enthusiastically 
recievednaiionallyand internationally, this series ot modules 
lollows the same basic lormat 

This module IS one ot a series ot competency *based guidance 
program training pachagestocusing upon spec itic professional 
and paraprotessional competencres ot guidance personnel 
The compeiencies upon which these modules are based were 
identitied and veritied through a project study as being those 
otcrtticalimportancetorthe planning supporting rmplemeni* 
ing operaiing and evaluating ot guidance programs These 
modules are addressed to professional and para protessionaf 
guidance program statt in a wide variety ot educaiionai and 
community settings and agencies 

Each module provides learning experiences that integrate 
theory and apptication each culminates with competency 
reterencedevaiuation suggestions The maienals are designed 
tor uSp by individuals or groups ot guidance personnel who 
are involved in training Resource persons should be skilled in 
the guidance program competency being developed and 
should be thoroughly oriented io the concepts and procedures 
used in the total training package 

The design ot the materials provides considerable ttexibitity 
tor plannrng and conducting competency-based preservice 
and mservrce programs lo meei a wide variety ot individual 
needs and mterests The materials are intended for use by 
universilies. state departments ot education, postsecondary 
institutions intermediateeducational service agencies. JTPA 
agencies employment security agencies, and other commu- 
nity agencies thai are responsible tor the employment And 
professional development of guidance personnel 

Thecompetency-ba:ted guidance program training oachages 
are products ot a research effort by the National Center s 
Career Development Program Area Many individuals, institu- 
tions and agencies pariicipaied with the National Center and 
have made contributions to ihe systematic devetopment 
tesiing and refinement of the materials 



National consultants provided substantial writing and review 
assistance in developmeni ottheininai module version^ over 
t300 guidance personnel used the matenatstn early stages of 
therr development and provided leedbach to the National 
Center lor revision and refinement The materials have been 
or are being usedby 57 pilot community implementation sites 
across the country 

Special recognition for mapr roles in the direction develop- 
ment, coordination ot development, testing and revrsion of 
these materials and the coordination ot pilot implementation 
sites IS extended to the following project staff Harry N Drier 
Consortium Director. Robert E Campbell Linda Ptister 
Directors. Robert Bhaerman. Research Spec^ahst Karen 
Ktmmel Boyle. Fred Williams. Program Associates, and Jame 
B ConnelK Graduate Research Associate 

Appreciation also is extended to the subcontractors who 
assisted the National Centenn this effort Drs Brian Jones 
and Linda Phillips-Jones of the American Institutes tor 
Research developed the competency base for the total pach* 
age managed project evaluation, and developed the modules 
addressing special needs Gratitude is expressed to Dr 
Norman Gysbers of the University ot Missouri*Columbia tor 
his worh on the module on individual career development 
plans Both ot ihese agencies provided coordination and 
monitoring assistance tor the pilot implementation sites 
Appreciation is extended tothe American Vocational Assocr 
ation end the American Association toi Counseling and 
Devetopmentforlheirleadershipin directing extremely impor- 
tant subcontractors associated with the tirst phase of this 
effort 

The National Center is graretui to the U S Oepartment ot 
Education. Of lice of Vocational and Adult Education (OVAE) 
tor sponsorship ot f hreecontraCts related to this competency- 
based guidance program training package In particular we 
appreciate the leadership and support ottered project staif by 
David H Pritchard who served as the project officer for the 
contracts We feel the investment of the OVAE m thistrarning 
package is sound and will have tasting effects m the field ot 
guidance tn the years to come. 

Robert E Taylor 
Executive Director 
National Center for Research 
in Vocational Education 
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ABOUT THIS MODULE 



Ootl 

After complattng this module, career guidance per- 
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DECISIONS priate audtehees and % u|e^^^ 

make key adintntstratlva dacisfona a^^^ 

gram and future direction, 

INTRODUCTION 5 

READING 7 

Competency 1. Clarify the decisions to which evaluation data may contribute and the 

characteristics and needs of various audiences interested In the evaluation results 7 

Competency 2. Choose a content, format, and level of sophistication for the evaluation 
report or presentation that is appropriate to the characteristics of the target audiences and 
their decision needs 8 

Competency 3 Develop rules to be followed in making program decisions, and clarify 

deci'5ions to which formative and summative evaluation data may contribute 9 

Competency 4. State the alternatives to be considered in making program decisions, use the 
decision-making rules and evaluation results to consider the various alternatives, and select 
the most beneficial alternatives for the program 10 

LEARNING EXPERIENCES 

1 Evaluating Results Who Needs Them and Why 11 

2 Communicating Results Effectively 15 

3 Deciding How to Use Evaluation Results 19 

4 Identifying Alternatives for Change 23 

EVALUATION 27 

REFERENCES 35 
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ABOUT USING THE CBCG MODULES 



CBCQ Module Organization 

The training modutes cover the Knowledge* skills* and 
attitudes needed to plan, support, implement, operate, and 
evaluate a comprehensive career guidance program. They 
are designed to provide career guidance program im* 
plementers with a systematic means to improve their 
career guidance programs. They are competency*t>ased 
and contain specific information that is intended to assist 
users to develop at least part of the critical competencies 
necessary for overall program improvement. 

These modules provide information and leaming ac- 
tivities that are useful for both school*baded and 
nonschool*based career guidance programs. 

The modules are divided into five categories. 
The GUIOANCE PROGRAM PLANNING category assists 
guidance personnel in outlining In advance what is to be 
done. 

The SUPPORTING category assists personnei in Know- 
ing how to provide resources or means that make it possi* 
ble for planned program activities to occur. 
The IMPLEMENTI NG category suggests how to conduct, 
accomplish, or carry out select career guidance program 
activities. 

The OPERATING category provides information on how 
to continue the program on a day-to-day basis once it has 
been initiated. 

The EVALUATING category assists guidance personnel 
in judging the quality and impact of the program and either 
making appropriate modifications based on findings or 
making decisions to terminate it. 

Module Format 

A standard format is used in all of the program's 
competency-based modules. Each module contains (1) an 
introduction, (2) a module focus, (3) a reading. (4) learn- 
ing experiences, (5) evaluation techniques, and (6) 
resources. 

Introduction. The introduction gives you* the module 
user* an overview of the purpose and content of the 
module. It provide enough inflation for you to deter- 
mine if the module addresses an area in which you 
need more competence. 

About This Module* This sectk>n presents the folk>w- 
ing intormation: 

IModulo Goal: A statement of what one can ac- 
complish by completing the module. 

Competencies; A listing of the competency 
statements that relate to the module's area of con- 
cern. These sta^ents represent the competencies 
thought to be most critical in terms of difficulty for 
inexperienced implementers, and they are not an 
exhaustive list. 

This section also serves as the table of contents for the 
reading and leaming experiences. 

Reading. Each module contains a section in which 
cognitive infonnation on each one of the competencies 
is presented. 

1 . Use it as a textbook by staning at the first page and 
reading through until the end. You could then 



complete the leaming experiences that relate to 
specific competencies. This approach is good H you 
vwuld like to give an overview of some competen* 
cies and a more in-depth study of others. 

2. Turn directly to the learning experienc€8(s) that 
relate to the needed competency (competencies). 
Within each leaming experience a reading is listed. 
This approach allows for a more experiential ap- 
proach prior to the reading acHvtty* 

Learning Eicperlenoee* The leaming experiences are 
designed to help users in the achievement of spectftc 
leaming objectives. One leaming experience exists for 
each competency (or a cluster of like competencies)* and 
each learning experience is designed to stand on ite own. 
Each leaming experience is preceded by an overview 
sheet which describes what is to be covered in the learn* 
ing experience. 

Within the body of the leaming experience, the following 
components appear. 

individual Activity: This is an activity which a person 
can complete without any outside assistance . All of the 
information needed for its completion is contained in 
the module. 

Individual Feedbactt: After each Individual activity 
there is a feedback section. This is to pn>vide users 
with immediate feedt>ack or ^valuation regarding their 
progress before continuing. The concept of feedback 
is alte Intended with the group activities* but it Is built 
right into the actMty and does not appear as a separate 
section. 

Qfoup Activity: This activity is designed to be 
facilitated by a trainer, within a group training session. 

The group activity is tdrmatted along the lines of a 
facilitator's outline. The outline details suggested ac- 
tivities and information tor you to use. A blend of 
presenlatk>n and "hands-on" participant activiee such 
as games and role playing is included. A Notes column 
appears on each page of the facilitator's outline* This 
space is provided so trainers can add their own com- 
ments and suggestions to the cues that are provided* 

Following the outline is a list of materials that will be 
needed by workshop facilitator. This section can senre 
as a duplication master tor mimeographed handouts 
or transparencies you may want to prepare. 

Evahiatlort TechnlqMet This section of each nwdute con- 
tains infonnatk>n and instruments that can be used to 
measure what workshop participants need prior to train* 
ing and what they have accomplished as a result of train- 
ing. Included in this sectkin are a Pre* and P^-Partlcipant 
Assessment Questionnaire and a Trainer's Assessment 
Questionnaire. The latter contains a set of performance 
indicators which are designed to detennine the degree of 
success the part'cipante had with the activity* 

Reterencea* All ma|or sources that were used to develop 
the module are Hated in this section. Also, major materials 
resources that relate to the competencies presented in the 
module are described and characterized. 



INTRODUCTION 



The basic purpose of evaluation is to provide 
Inlorntatlon tor planning and future action or 
decisions. A program evaluation seeks to deter- 
mine whether or not the program activities have 
met their objectives. Practically, this means. *'Did 
your clients learn what you intended them to 
learn?" Realistically, these results often have 
substantial impact on both the direction and life 
of your program. And. in this age of accountabih 
ity. many audiences are interested in the findings. 
These audiences range from your clients and 
parents or guardians (where appropriate) to tax* 
payers and legislators**particularly if your pro* 
gram is being supported with stale or federal 
funds. 

Evaluation is a Hey component in any career guid- 
ance program, not an add*oa Although this 
module and the module CG E*1. Evaluate Guid- 
ance Actmttes. appear to complete the series, 
they are topics and competencies that must be 



undertaken as you plan your program. The care* 
ful setting of objectives, making sure that they are 
both important and measurable, will ensure that 
you obtain information to refine and improve 
your efforts. 

Communicating clearly is important in all aspects 
of program development and implementation. II 
is particularly essential to keep people up to date 
with the progress your program is making in 
reaching its goals. This module focuses on how 
to communicate this progress as well as how to 
use evaluation findings to improve your program. 
II ties not only with the module CG E*1. fva/uafe 
GuidBnce Activities, but also with the first module 
of the series, CG A-1 . Identify and Plan for Guid- 
ance Program Change, thus illustrating the cycli* 
cal nature of program development and 
implementation. 



READING 

Evaluating Results: Who Needs Them and Why 




There are many people who believe that evalua* 
tion reports are never read--only tited. Unfortu- 
nately, it does seem that all too otten an evalua* 
t(on is conducted to meet the demands ot a 
funding agency and critics justitiably claim that 
the report is simply a product to meet contract 
requirements However with increasing demands 
on limited tunds and heightened citizen concern 
tor use of tax monies, more and more people are 
interested in whether or not the program that you 
provide has 9 positive impact on your clients. 

As you plan to communicate evaluation results to 
your constituents, you need to identify the var* 
lou6 audiences and determine what information 
they need for decision making. If you are receiv* 
ing external funding lor your program, these 
sponsors are* of course, a key audience. But. who 
else needs information? Whatever the structure 
of your agency is. certain administrators will need 
evaluation information. If you have an advisory 
committee (several members of whom might 
serve as an evaluation team), you will want to 
keep them advised of both progress and prob* 
lems As both staff members and clients will 
undoubtedly be involved in any modification of 
the program— as well as possible providing the 
basic evaluation data— they should also be viewed 
as key audiences One often overlooked group is 
the community The public i$ more interested 
than we often credit them for in the results of our 
programs. It is often their tax dollars and their 



education orthatof their children that are affected. 
In addition, many career guidance programs rely 
on the participation of the community to be truly 
elfective. Depending upon the impact of the eval* 
uation there may be additional audiences such as 
legislative leaders or board of education members 
that need to be advised of the findings. Also, be 
sensitive to any ''right*tO'know" audiences**that 
is. anyone who would be affected by the results. 

Evaluation data are generally used to make one or 
both of the following decisions: 

1 How can this program be improved? 

2. Should this program be continued? 

The first question is usually the focus of a forma* 
tive evaluation; that is« a study of the process of 
program implementation. Such data are of par* 
ticular use to program administrators as they 
work to improve particular components of the 
program. The second question is, of course, more 
consequential to the life of the program. This 
decision can be made by data gathered through a 
summative evaluation study. Summative evalua* 
tions are results oriented and seek to find the 
effects that program efforts have on clients. A 
variety of audiences are interested in these find- 
ings because the resulting decision defines the 
worth of the program. 
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Communicating Results Effectively 



The importance of being sensitive to the individ- 
uality of your audiences is easy to acknowledge 
but somewhat more complex if you intend to 
carry it out fully. There are many things to con- 
sider how much vIsiWIy should this program 
evaluation receive: how many dillerent audiences 
must the results reach; how much time and 
money should be devoted to disseminating the 
research findings before beginning to make pro- 
gram changes? In essence, communicating eval- 
uation results is important; however, the extent of 
the dissemination process must be thought through 
carefully 

Your communication strategies will differ depend- 
ing upon the type of evaluation resutts you have 
to report Formative evaluation results which 
address program components on an ongoing 
basis often are directed toward fewer audiences 
than are the summative evaluation results and are 
usually presented more informally and provide 
suggestions that are useful primarity to program 
staff Summatlve evaluations, however, often are 
of interest to a wider variety of audiences as they 
reflect end-of-program results- The reporting of 
these results can take a variety of forms, but 
should include the following elements: 

1 Introduction A statement of purpose and a 
brtof description of the program should pro- 
vide sufficient background for the reader to 
make use of the study*s recommendations, 

2 Recommendations. The reader should know 
what recommendations have been made 
prror to readingthedetailsofthe study. This 
IS the most important section of the report, 
Rrcommendations should be clear and con- 
cise and directly related to the findings of 
the f*tudy That is, if there are five major 
recommendations, there should also be five 
sets of findings to support those recom- 
mendations From this, program staff and 
others can suggest methods to act on the 
recommendations 



3, Evaluation Design and Procedures. This 
section should describe the design, the 
instrumentation, data sources, and any lim- 
itations within the study Although this part 
of the report will undoubtedly be quite tech- 
nical, present your techniques as simply 
and concisely as possible. There are always 
several approaches that could be used in 
designing an evaluation study. Be thorough, 
but don't "over justify'' your design. 

4, Findings, Use simple charts and tables o 
present your results. As you are reporting 
results—not making recommendations- 
refrain from expliitining the results. Also, 
present your data m the most useful way 
possible For example, use percentages such 
as ''85 percent agree and 15 percent dis- 
agree'' as opposed to using raw numbers. 

Some evaluators believe that the same report 
should be provided to all audiences with an exec- 
utive summary listing Hey findings and recom- 
mendations for those who are either not inter- 
ested or sufficiently skilled to analyze the full 
report. This approach minimizes the expense of 
communication and avoids any misinterpreta* 
tions that staff are providing different information 
to different audiences. Another approach—more 
widely approved— is to develop adlflerent report, 
as well as a different style of presentation, for 
each audience. In adopting this approach, you 
should receive more attention from the audiences 
you wish to reach. 

If you decfde that your evaluation report should 
receive wjde dissemination and that the implica- 
tions of the findings and recommendations merit 
interaction with various audiences, you may wish 
to use means jther than written reports to reach 
even more people. Use of the news media- 
newspapers, radio, and television— provides im- 
mediate visibility This type of coverage can serve 
as a lead-in to a series of meetings to discuss the 
study in detail 
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Deciding How to Use Evaluation Results 




The true impact of an evaluation depends on 
whether or not the results are used to improve 
programs. A great deal of data are generated in 
most evaluations—some positive, some negative-- 
and it may be your job to sort out the information 
and make sense of it. Although the data reveal to 
you how clients perform, you and others have to 
set the standards to interpret the results accu* 
rately For example, if you use portions of instru* 
ments to measure how well your clients achieved 
a particular objective, what percentage of achieve* 
ment will prove attainment— 50 percent* 75 per- 
cent. 90 percent? In short, how well do your 
clients have to perform in order for you to be 
satisfied that they have met the program goals? 

It is likely that as you examine the data, you will 
find that some obfectives were clearly met. some 
were clearly not met, and some fall in between. To 
deal with these sorts of mixed llndlngst this set of 
guidelines can be used to analyze the results; 

1 Examine all of the program objectives and 
separate your llndings into two areas: those 
that met the criteria for succer.s and those 
that did not^ 

2 Analyze the program objectives that were 
met Are there clues as to why they were 
achieved? Could activities be improved to 
either reach more clients or be more cost* 
effective'' 



3. Analyze the program objectives that were 
not achieved. Are these objectives realistic? 
Should they be revised or discarded? Or* 
should other activities be undertaken to 
meet the objectives? 

4. Determine a revised set of objecDveSt per- 
haps including some objectives that were 
not reached, 

5. Plan activities which are likely to achieve the 
desired outcomer 

Using this framework, or a similar one. can help 
you in analyzing results from either formative or 
summativeevaluationsas theprocess for making 
decisions does not change. Decisions resulting 
from formative evaluations are incorporated into 
the program as it is developing. If it is discovered 
that particular objectives are not being achieved* 
activities may be changed or new ones imple- 
mented. Decisions madelrom summative evalua* 
tions usually involve more extensive modifica- 
tions such as continuing or discontinuing certain 
program objectives. 

As you begin planning for program modifica* 
tions. be sure to look beyond the recommenda- 
tions listed in the report. Evaluate the evaluation 
itself to make certain that the decisions you make 
for change have sufficient data to support the 
actions you take. 
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Identifying Aiternatlves for Change 



>$mBJI^^^^ in m«Klng 

ii ui* tH« d0citton*mftkini$ ruit» 



h is rareloever find an evaluation report that does 
not make recommendations for program improve- 
ment This does not mean that most programs are 
of poor quality nor does it reflect on the useful- 
ness of evaluation, but rather on the vast number 
of objectives that are present in most programs— 
and the near possibility of perfection' 

When conclusions are offered as to program 
effectiveness, there is a variety of allernallves that 
are available to improve the effort, Even if a pro- 
gram meets all of its objectives, alternatives may 
need to be examined to see if there areother ways 
to meet the objectives and yet reach additional 
clients-perhaps wMhout using additional funds. 

On the other hand, if a program meets none of its 
objectives, there are alternatives to consider 
besides discarding the entire effort. Would the 
program have worked if there had been additional 
resources to support it? Were the objectives 
attainable or. with revisions, are they stilt 
importanf^ 

Often the objectives we establish for programs 
are a setup for failure because they are so idealis- 



tic that they are nearly impossible to attarn> They 
should be reexamined before final decisions are 
made as to the future of the program. 

What typically appears in evaluation findings is 
something less dramatic than total success or 
total failure. Regardless of the positive or nega- 
tive finding, you should examine each objective 
and the activities undertaken to achieve it as out- 
lined in the previous section. Analyze the data 
thoroughly before generating alternatives for 
change. In addition, it is important to analyze how 
ready your organization is to undertake change. 
If resistance is likely to be met regardless of the 
content of the change, work must begin there. 
You cannot implement change within your agency 
by yourself or with a small staff. It takes the com- 
mitment of the administration and staff, and this 
is more likely to occur by involvement rather than 
mandate. In essence, the same process that you 
undertook to initiate the program—planning, 
supporting, implementing, operating, and 
evaluating— must once again be undertaken as 
you make program changes. 
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Learning Experience 1 

Evaluating Results: Who Needs Them and Why 

OVERVIEW 



C0W«I^GTENCY Ct^rtfy the docto^brifd to wh'^^ evaluation data may contribute 

^inta fl^,\dMftr^ctertftt4c9.dnd noM$ of various audiences inter- 
' tow th«evftiu<iU4>if1 results. 



INDIVIDUAL 
ACTIVITY 



INDIVIDUAL 
FEEDBACK 



INDIVIDUAL 

LEARNING 

OBJECTIVE 



■;('■ t'. 



iist,the |Mdtttriccta to be served.through evaluation ellortt^. 



Identify the various audiences to whom you plan to distribute 
your evetuation f ini^inss, 



Compare your ideas witn notes provitjed. 



GROUP 

. pADMiu A Verbalize the potential Impact that evatufition results have on 

L&AnNInu various deci^on'maKing atidlences, 

OBJECTIVE 



GROUP Identify decisions that will be made as a result of a program 

ACTIVITY evaluation. 



1 1 



INOlVlOUAi. ' Idtntlfy the vtrtoiM audllncM to whom you plan to dlttrlbute 

ACTIVITV ' ■ 



Review the reading for Competency 1 on page 7, and then complete the following activity: 

As director of your agency's career guidance program, you have decided that-after 3 years-it is 
essential that a comprehensive summative evaluation be performed. Your program's efforts have been 
guided by detailed objectives anr' formative evaluations have been conducted periodically. 

Your program has had high visibility both Inside and outside your agency. List those audiences whom 
you would communicate findings and rank them in order of importance. 

Audiences Rank 
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INDIVIDUAL 
FEEDBACK 



Cbmpar* your l<^M 




Did you Mst the following audiences? 
Yes No 



t Funding Sponsor 

2. Advisory Committee 

3. Board of Education/Board of Directors 

4. Advisory Committee 

5. Staff (including adminstrators) 

6 Clients 

7 Parents or Guardians (if appropriate) 

8 Community Leaders 

9. legislators 

10. Professional Colleagues 



In ranking fnr importance, there is one "right** answer However, your listing will indicate your own 
values and may reflect the audience(s) to whom you are most responsive. 



Note: The following outline is to be used by the workshop facilitator. 



A Introduction 

1. Indicate that this activity is designed 
to aid participants in identifying 
decisions made by various groups 
as a result of program evaluation 
data 



GROUP 
ACTIVITY 




Fadlltator's Outline 



Notes 



ERIC 
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FactUlalor^t OulUne 



Notea 



P Explain that this activity requires 
participants to role play in small 
groups. 

3. Have participants review the read* 
iiig for Competency 1 on page 7. 

B Process 

1 Ask participants to break into three 
groups. 

2 Assign each group a simulated iden* 
tification label such as: (1) agency. 
(2) staff, or (3) community. Ask par- 
ticipants to assume the roles of 
representatives of these audiences. 

3. Have participants assume that they 
have received an interim evaluation 
of a career guidance program indi- 
cating positive evidence of change 
in the types of learning activities 
provided forclients* but no evidence 
of significant change in the clients* 
knowledge or skills. 

4. Have each group list ways In which 
their decision*making roles could 
impact the future of the program. 

5. Reassemble as a total group and 
share findings, comparing and con- 
trasting decision* making powers of 
each group. 



Encourage gt uups to go beyond obvious actions 
such as eliminating funding to listing positive 
actions they could take. 
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Learning Experience 2 

Communicating Results Effectively 

OVERVIEW 



COMPETENCY 



Choose a content, format, and leval of sophistication for the 
evaluation report or presentation thai is appropriate to the 
Characteristics of the target audiences and their decision 
needs. 



READING 



Bead Competency 2 on page 8. 



INDIVIDUAL 
ACTIVITY 



Designs fating form toevaluatethe content and format of your 
evaluation report- 



INDIVIDUAL 

LEARNING 

OBJECTIVE 



t'rapare a rating form to evaluate the content and format of 
your evaluation effort. 



GROUP 
ACTIVITY 



Check your form with the sample provided. 



GROUP 

LEARNING 

ACTIVITY 



De^gn a plan for communicating your evaluation results. 



GROUP 
ACTIVITY 



Discuss various approaches for communicatin 3 your evalua- 
tion results. 



IS 



ERIC 



liHilMitt 'OfW to •«»u«te tha content and format of 

t:^-'-" -rt . ^ ^ :" , , ■ " r: ■ ^ . . _ \ : 



As you review the reading for Competency 2 on page 8. you will note that your basic elements of a 
report are listed 

1 Introduction 

2 Recommendations 
3^ Evaluation Design and Procedures 
4 Findings 

■ Prepare a form that lists the criteria, components, etc.. you would look for within each element of a good 
evaluation report. In other words what would be the review criteria you would use to measure the 
quality of a report. 



ERLC 
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INDIVIDUAt 
FEEDBACK 



As you compare your form to the sample provided, note that by using the categories '*weii stated/' 
"needs better statement/* and "not stated/' you can evaluate not only whether or not an element was 
included* but also how well it was presented. 



Sample Evaluation Report 
Rating Form 



Introduction 

1 Purpose of study 

2 Program objectives 

3 Program activities 

4 Client description 



Well 
Stated 



Needs 
Better 
Statement 



Not 
Stated 



Recommendations 

1. Related to findings 
2 Viability 



Evaluation Design and Procedures 

1 Framework explanatfon 

2 Data collection procedures 

3 Analysis process 



Findings 

1 Data presentation 

2 Accomplishment of objectives 

3 Otitcomp^ summary 

17 
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Note: The following outline is to be used by the workshop facilitator. 



Facilitator's Outline 


Notes 


A Introduction 




1 


Explain that the purpose of this 
activity is to discuss effective ways 
to communicate evaluation findings. 




2 


Discuss with participants items tor 
consideration when disseminating 
findings, e.g.. number of audiences, 
time and funds available, need for 
visibility. Use the reading for Comp- 
etency 3 on page 9, as a basis for the 
discussion. 




B Process 






Have participants list possible dis- 
semination techniques (e.g., format 
report, executive summary, separate 
reports, news releases, audiovisual 
presentations). 




2 


Ask participants to discuss strengths 
and weaknesses of each strat?gy. 




3 


Have participantswrite brief outlines 
of their preferred communication 
design^ 




4. 


Ask participants to share the ration- 
ale for their decisions. 
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Learning Experience 3 

Deciding How to Use Evaluation Results 

OVERVIEW 



COMPETENCY 



READING 



INDIVIDUAL 

LEARNING 

OBJECTIVE 



INDIVIDUAL 
ACTIVITY 



INDIVIDUAL 
FEEDBACK 



GROUP 

LEARNING 

OBJECTIVE 



GROUP 
ACTIVITY 



OevQlopfui0$ to b^fotiovt^iAinaKIng program decisions, and 
olarlly declsilons to which formative and summattve evaluation 
data nf»ay <s<mtrit)Mte, 



Rea^ Corhpetency 3 on t>a^e 9. 



petermlrte art appropriate procesi^ to analyze pt'ooram^valua- 

■tionresMNs.,:"-: " 



Design a ruet^od by. which a $roupoan examine avaluatlon 
Insults and make progam declstons- 



Compare your ideas with the notea ahd sampfe chart |>roytded. 



Discuss the utility of formative and summeitive evaluation deta 
in making program decisions. 



Brainstorm questions that can and cannot be answered as a 
result of formative and summative evaluations. 
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iMpWtDIIA^ , 0«f iQn « mattiod by which « group can examine evaluation 

Jj^ff^^ '**^lta and make program 



Before you start this activity, review the reading for Competency 3 on page 9. You have been assigned 
the responsibility of chairing a task force of agency and community representatives whose purpose is 
to make recommendations for improving your agency's career guidance program. A summative 
evaluation has recently been completed, and you and other members of the task force have received the 
completereport. How would you plan to lead the group in analyzing the results? Remember--notallof 
your task force members are educators, nor are all educators evaluation experts. 
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INDIVIDUAL 
FEEDBACK 



It is important that you make it as easy as possible for people to use the data listed in the report. The 
recommendations should serve as support but should not limit the group's actions. 

As was listed in the reading, look initiallyat the objectives that were accomplished and those that were 
not. From that point, determine what activities affected the achievement or lack of such before having 
the group suggest changes. If the process is carried out in a step-by-step manner, confusion- and 
overreaction-should be kept to a minimum. A sample analysis chart follows. 



Career Guidance AdivHies Evaluation 
Analysis Chart 



Grade Level Tatclisr/Couruelor Caresr Oevalopment Arsa 

Age Level 

Oevelopmental Goal 





Time 


Cost 


No, of 

Resource 

Penont 


■ 

No. of 
CItenti 


Activity 
Ostet 


Career 

Oomaint 

Oevelop- 

ment 

Needs 


Comments 


Objective 1 

Measurement Degree of 
Date Success 
















Activity 1. 
















Activity 2. 
















Activity 3. 


1 




1 




i 
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Note: The following outline is to be used by the workshop facilitator. 



FaOIHators Outline 


Notes 


A. Introduction 




1. Explain that the purpose of this 




activity is to aid participants in 




determining types of questions that 





can and cannot be answered through 
formative and summative evaluations. 

2. Indicate that this activity will require 
participants to be involved in brain* 
storming in one of two groups. 

3 Have participants review the read- 
ing for Competency 3 on page 9. 

B Process 

1 . Divide the participants into groups- 
Formative and Summative- 

2. Have each group brainstorm ques* 
tions that can be answered in using 
that particular type of evaluation. 

3. After each group has a list of ques* 
tions. have them list the types of 
data that would need to be gathered 
to answer each question 

4^ Reconvene as a total group and 
share problems that occurred in the 
process, having members of the 
opposite group assist in determin* 
my ways to obtain data for the 
unanswered questions. 



22 



2-1 



You may wish to note that some of the same 
questions may be raised by both groups. 



Inform participants that they can eliminate 
questions for which they have no feasible way 
of gathering data. 



Learniirig Experience 4 

Identifying Alternatives for Change 

OVERVIEW 



COMPETENCY 



state the alternatives to be considerecl th making program 
decisions, use the decislon*making rules and evaluation .'esults 
to consider the various alternatives, arid select the most bene- 
ficial alternatiyes^or the program. 



READING 



Read Competency 4 on page 10. 



INDIVIDUAL 

LEARNING 

OBJECTIVE 



state alternatives to be considered in making declsforts about 
program modiflcaiiOns. 



INDIVIDUAL 
ACTIVITY 



List concerns and alternatives regarding pfiC^ram changes to 
be considered In expanding your program ; 



INDIVIDUAL 
FEEDBACK 



Compare your notes with the statements provided. 



GROUP 

L PARMIMfi Provide a rationale supporting your recommendation to con- 

T^^"'?!?'** tinue or discontinue a program. 

OBJECTIVE; 



GROUP Analyze results of program evaluations and make decisions as 

ACTIVITY whether or not they should be continued. 




IK - •H«rni1lvtt r»oirding prograrn changes to 



fleviewthe reading for Competency 4 on page 10, You have just received the good news-Mhe positive 
evaluation results of your '^moder* program have reached your chief executive, and you have been 
given the responsibility to expand your program^-as is-throughout your agency within the next 6 
months What immediate concerns arise in your mind, and what alternatives do you visualize that might 
alleviate them'^ 
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INDIVIDUAL",'.' ' ' 'Qmw»-ii«ir»it»Mm(i^ ili^'kiiM^' 



Regardless of the type of program or the client group(s) it serves, certain concerns probably come to 
your mind. As tnere was no mention of increased funding in the plans to expand yourprogram, this may 
be your first concern ^ven if a substantial amount iS allocated, it is likely that you will not have the same 
dollars per client to use as you did in your model program. Before attempting to replicate your program, 
you might be advised to look at ways in which activities can be accomplished at a lower cost. 

No mention was maoeof the objectives in your program that were not met-and there undoubtedly were 
some. Analyses of these objectives and their related activities should be undertaken at once before 
expansion begins. 

You also may have concerns about implications for increased staffing needs, additional materials to 
reach a larger range of clients, and other problems that arise when providing a program that serves a 
wide range of audiences. Advise othersof these implications, too, so that a later evaluation can reflect 
the major changes that have been undertaken. 



GROUP 
ACTIVITY 




Note: The following outline is to be used by the workshop facilitator. 



Facilitator's Outline 


Notes 


A Introduction 




r Indicate that the purpose of this 




acdvity is to apply participants' 




learnings to **real*life** situations. 




2. Explain that the success of this 




activity depends upon the involve- 




ment and sharing of group members. 
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FacMitator'f Outline 


Notes 


3. 


Have participants review the read- 
ing tor Competency 4 on page 10. 




B. Process 




1 


riences of program evaluations and 
the decision alternatives that were 
available as a result. 


You may wish to provide the first example as a 
model. 


2, 


Invite participants to ask questions 
of the ''presenter" to gain additional 
information as to the details of the 
program and the evaluation process. 




3, 


Have participants voteas to whether 
or not they would continue the 
program. 




4. 


Ask participants to justify their deci- 
sions, and invite the ''presenter*' to 
react. 




5. 


Repeat the process with two to three 
additional presentations. 
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EVALUATION 



PARTICIPANT SELF-ASSESSMENT QUESTIONNAIRE 



1 N^inM' fOptionnn 

Agency Setting (Circle the appropriate number) 



6 ErMiionlafy School 

7 /mrtxify School 

ft Pn*K<^roficla»y School 

9 CdiiociP UntvofSiiy 



10 JTPA 

1? Chufch 
13 CoTrections 



15 
17 



3 Dale 
4 Module Niimhtr 



Youm Services 
Business industry 
Manage me n1 
Business Industry Labor 
Parent Qroup 



18 Miinicipal Office 

19 Service Organiiation 
70 Slale Oovernnnent 
?1 Other 



Workshop Topics 



PREWORKSHOP NEED FOR TRAIN- 
ING Degree of ^ecd (circle one for 
each workshop topic) 



POSTWORKSHOP MASTERY OF 
TOPICS Decree of Mastery (circle 
one for each workshop topic). 



///// ///// 



1 Listing the audiences to beserved through 
evaluation efforts for disseminating eval- 
uation reports to a variety of audiences. 

2 Verbalizing the potential impact that eval- 
uation results have on various decision- 
making audiences. 

3 Designing a rating form forev^'uating the 
evaluation report 

4 Designing a plan for communicating eval- 
uation results. 

5 Determining an appropriate process to 
analyze program evaluation results. 

6 Discussing the utility of formative and 
summative evaluation data in making 
program decisions. 

7 Stating alternatives to be considered in 
making decisions about program 
modifications 

8 Providing a rationale supporting your 
recommendation to continue or discon- 
tinue a program 



0 12 3 4 



0 12 3 4 



0 1 

0 1 

0 1 

0 1 



2 3 4 

2 3 4 

2 3 4 

2 3 4 



0 12 3 4 



0 12 3 4 



0 12 3 4 



0 12 3 4 



0 1 

0 1 

0 1 

0 1 



2 3 4 

2 3 4 

2 3 4 

2 3 4 



0 12 3 4 



0 12 3 4 



Overall Assessment on Topic of Communi- 
cate and Use Evaluation-Based Decisions 



3 4 



12 3 4 
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Comments: 



Trainer*8 Assessment Questionnaire 



Trainer: . . . , 
Title of Module: 



Date: 



Module Number: 



Traming Time to Complete Workshop: 

Parttcipant Characteristics 

NumberinGroup Number ol Males. 

Dtstributton by Posttion 

Elementary School 

Secondary School 

- --- Postsecondary School 

_ . . College/University 

JTPA 

Veterans 

Church 

Corrections 



hrs. 



mm. 



Number ol Females 



Youth Services 

Business/Industry Management 

Business/Industry Labor 

Parent Group 

Municipal Office 

Service Organization 

State Government 

Other 



PARTi 

WORKSHOP CHARACTERISTICS— Instructiont: Please provideany comments on the methods and 
materials used* both those contained in the module and others that are not listed. Atso provide any 
comments concerning your overall reaction to the materials* learners' participations or any other 
positive or negative factors that could have affected the achievement of the module*s purpose. 

t Methods: (Compare to those Suggested in Facilitator's Outline) 



2 Matertals: (Compare to those suggested in Facilitator's Outline) 



3 Reaction: (Participant reaction to content and activities) 
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PART II 



WORKSHOP lMPACT~lfi8truc1IOfii: Use Performance Indicators to judge degreeof mastery (Com- 
ptete responses for all activities. Those that you did not teach would receive OJ 

Group's Decree of Mastery 



Not 
Taught 



Utile 



Some 



Good 



Outstanding 
(over 75^) 



Mole: CIrele the number that best retlecis your opinion ol group mastery* 



Learning Experlvnce 1 








Group 


0 1 


2 


3 4 


Individual 


0 1 


2 


3 4 


Learning Experience 2 








Group 


0 1 


2 


3 4 


Individual 


0 1 


2 


3 4 


Learning Experience 3 








Group 


0 1 


2 


3 4 


Individual 


0 1 


2 


3 4 


Learning Experience 4 








Group 


0 1 


2 


3 4 


Individual 


0 1 


2 


3 4 



Code: 

Liniet With no concern tor time or circumstances within training setting it it appears that less than2S%ot the learners 
achieved what was mtended to be achieved 

Som^ With no concern tor time or Circumstances Within the training setting it it appears that less than close to h alt ot 
the learners achieved the learning experience 

Good: With no concern tor time or Circumstances With in the training Setting It It appears that 50%-75% have achieved 
as expected 

OulSlanding: tf mot^^ than 7S% ot learners mastered the content as expected 
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PART III 



SUMMARY DATA SHEET— Inslructlons: (n order to gain an overall idea as to mastery impact 
achieved across the Learning Experiences laughli complete the following tabulation. Transfer the 
number lor the degree of mastery on each Learning Experience (i.e.i group and individual) from the 
Workshop Impact form to the columns below. Add the subtotals to obtain your total module score. 

GROUP INDIVIDUAL 

Learning Experience Learning Experience 

1 * score (1-4) 1 = score (1-4) 

2 score (1-4) 2 - score (1-4) 

3 ■ score (1-4) . 3 = score (1-4) 

4 - score (1-4) 4 = score (1-4) 

Total Total 

(add up) (add up) 



Total of the GROUP learning experience scores and INDIVIDUAL learning experience scores ~ 
- Actual Total Score Compared to Maximum Total* 

'Maximum total is the number of learning experiences taught times tour (4). 
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NOTES 
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Performance Indicators 



As you conduct the workshop component ot this training module, thetacilitator's outline will suggest 
individual or group activities which require written or oral responses. The following list ot performance 
Indicators will assist you in assessing the quality ot the participants* work: 

Module Title: Communicate and Use Evaluation-Based Decisions 
Module Number: CG E-2 



Group Learning Activity 



Performance Indicators to Ba Us«d for Learner Assessment 



Group Activity Number 1: 

Identity decisions that will be made 
as a result ot a program evaluation. 



1. Can participantslist three audiences tor evaluation reports. 

• Funding agency 

• Statt 

• Community 

2. List kinds ot things each group would like to Know as a 
result ot an evaluation, 

3. List two decisions which each group might make as a result 
ot an evaluation report. 



Group Activity Number 2: 

Discuss various approaches for 
communicating evaluation results. 



1. List items to be considered prior to disseminating evalua* 
tion tindings. 

• Number ot audiences 

• Time and tunds available 

2. List dissemination techniques, 

• Formal report 

• Executive summary 

• News releases 

• presentations 



Group Activity Number 3: 

Brainstorm questions that can and 
cannot be answered as a result ot 
tormative and summative 
evaluations. 



1^ Can participants articulate the ditterence between torma- 
tive and summative evaluation. 

2. Examinelistot questions which tormative evaluation answers. 

3. Examine list ot questions which summative evaluation 
answers, 

4. Can the participants identity the data sources needed to 
answer the above questions? 
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Group Learning Activity Performance Indicators to Be Used for Learner Assessment 



Group Activity Number 4^ 

1 Given an evaluation report, the group should be able to 
identify program strengths and weaknesses. 

2. Follow the logic to see it solid progra*^ modifications can 
be suggested as a result of evaluation report 

3. Shouldcertaincomponents be discontinued? Which ones? 
On what basis? 



Analyze the results of program 
evaluations and making decisions 
whether or not they should be 
continued 
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Ms Grace Baivnssr 
Pist Prendsftt 
NatiOftii FsrsnhTeschsr 
Association 

0' FrsnhBowe 

Former Executive Director 

Ms Jane Rsteohi 

£docat»on Coordtrtstor 
Amsficsn Coalition of Citizens 
with Oissb4litle$ 

Mr Robert L Crsio 
Vtts Presidsnt 

Qovemmsfil artd Pvblic Affairs 
Ainbricsn Society lor Tramino 
and Devslooment 

Or Walter Oav4s 

Oi reel or of Education 

AFL CIO 

Or Rtchard OiEugenio 
Senior Lesislsitve Associaie 
(rspr^ssntino Conore$s«nan Bill 

Qoodliits) 
House Education and Lat>or 

Committee 

Mr Oscar Qie^nes 
Administrator (Reh'ed) 
US Departmeni of Labor 
Division of EmcHoyment artd 
Trsinir»9 

Or RobsrtW Qtovsr 
Director artd Chatrpsrson 
Fedefa' Commiltss on 

Apprertiiceship 
The Univsrsity of leiias at Austin 



Or. Jo Hayslip 
Erector of Ptann4r»9 artd 

Oeyelopment tn Vocational 

RaKabllltatlon 
New Hampfhlra State Department 

of Educeiion 

Mrs. Madeleine Hemmlnos 
National AlUsitce lor Business 

Dr. Edwin HOrr 
Coufiaefor Educator 
Pennsylvinie State Universtly 

Or Elaine House 
Proiesaor Smehtus 
Ruleers University 

Or Oavid Lacey 
Vice president 

Personnel Ftarmino and Business 

Inteoratlon 
aONA Corporation 

Dr. Howard A Mattttews 
Asakstarvt Staff oireclor 
education (representing Senator 

OrinO Hatch) 
Committee on Labor and Human 

Resources 

Dr. Lee McMurrHi 
Superintendent 
MiiwaiikeePuMic Sc^ooia 

Ms, Nantrte Meiiiie(ohn 
Assistant Director of Legislation 
American federation of State. 
County, and Municipal Employeee 



Or. Joseph 0- Milts 

Slate Director of Vocational 

Education 
Florids Department of Education 

Or. Jacl( Myers 

Director of Heaim Policy Study and 

Pri¥ite Sector Inittstive Study 
American Enterprise Instituie 

Mr^ Reid Rundeil 

Oirector of Personnel Deveiopmervt 
Oenenl Motors Corporation 

Mrs. Dorothy Shields 
tducetion 

American federation ol Labor/ 
Conoresa of Industrial 
Oroanitations 

Or. Barbara Thompson 
former State Superintendent 
Wisconsin Department of Public 
Instruction 



Director 

Employmarrt and Training Division 
National Oovernors* Association 

Honorable Chalmers P Wylie 

Conoressman/Ohio 

U.S. Congress 
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